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Welcome 


to  this  edition  of  the  MSIAC  Journal.  Increasingly,  individuals,  groups  and  communities  across 
DoD  are  looking  to  M&S  to  improve  their  functions  and  business  processes.  Some  organizations  are  expanding  their 
investments  in  new  and  emerging  technology.  Some  are  developing  innovative  applications  for  M&S,  while  others  are 
trying  to  understand  the"whys  and  wherefores"  of  M&S.  All  have  a  common  approach:  they  are  seeking  solutions  and 
exploring  possibilities  through  the  "lens  of  M&S." 


This  issue  of  the  MSIAC  Journal  examines  these  three  separate  viewpoints:  M&S  technology  development,  M&S  application, 
and  the  understanding  of  M&S  fundamentals.  George  Mason  University's  Volgenau  School  of  Information  Technology  and 
Engineering  presents  an  approach  to  and  the  results  of  experiments  examining  how  best  to  utilize  automated  geospatial 
tools.  The  article  by  the  Proteus  Management  Group  explores  applying  gaming  technology  to  examine  the  volatility, 
uncertainty,  complexity,  and  ambiguities  of  evolving  warfare  techniques.  The  paper  by  Mr.  Randy  Koons  of  Alion  Science 
and  Technology  demonstrates  how  media  packaging  and  our  chronological  age  affects  the  utility  of  training  systems  and 
their  development. 

I  hope  that  you  enjoy  reading  this  selection  of  articles  and  papers  on  different  viewpoints  through  the  "lens  of  M&S." 


Dane  Mullenix,  MSIAC  Director 
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Evaluation  of  Advanced 
Automated  Geospatial  Tools 


Center  of  Excellence  in  C41 
TheVolgenau  School  of  Information  Technology  and  Engineering 
George  Mason  University 


By:  Kathryn  Blackmond  Laskey 
Walter  A.  Powell 
Leonard  Adelman 
Michael  Hieb 
Martin  Kleiner 


Abstract 

Geospatial  reasoning  has  been  an  essential  aspect 
of  military  planning  since  the  invention  of  cartography. 
Although  maps  have  always  been  a  focal  point  for  devel¬ 
oping  Situational  Awareness,  the  dawning  era  of  Network 
Centric  Operations  brings  the  promise  of  unprecedented 
battlefield  advantage  due  to  improved  geospatial  situ¬ 
ational  awareness.  Automated  Geospatial  Tools  (AGTs) 
are  ubiquitous  within  current  military  forces  and  also 
civil  and  humanitarian  organizations.  Nevertheless,  there 
is  too  little  empirical  evidence  to  quantify  the  military 
value  of  automated  geospatial  tools  to  the  warfighter.  As 
research  and  development  efforts  progress  to  bring  forth 
the  next  generation  of  AGTs,  Advanced  Automated  Geo¬ 
spatial  Tools  (AAGT),  it  is  vital  to  inform  the  development 
process  with  sound  empirical  assessments  of  the  military 
value  of  AAGTs  within  a  Network  Centric  Environment. 
To  this  end,  the  U.S.  Army  Topographic  Engineering 
Center  (TEC)  is  sponsoring  a  series  of  experiments  to 
evaluate  the  value  of  its  AAGT,  the  Battlespace  Terrain 
Reasoning  and  Awareness  Battle  Command  (BTRA-BC) 
Tool.  This  paper  discusses  the  scope  of  the  current  experi¬ 
ment,  the  hypotheses  we  intend  to  investigate,  and  the  ex¬ 
perimental  design.  Results  of  our  first  set  of  experiments 
will  be  provided  at  the  conference  and  in  a  later  version 
of  the  paper. 


1.  Overview 

The  focal  point  of  the  battlefield  command  post  is 
the  map.  Through  interactions  with  the  map,  the  com¬ 
mander  and  staff  collaborate  to  build  a  common  operat¬ 
ing  picture.  This  common  operating  picture  displays  the 
area  of  operations,  the  militarily  significant  features  of 
the  terrain,  the  locations  of  adversary  friendly  forces  and 
neutral,  and  the  evolving  plan.  A  generation  ago,  planning 
centered  on  a  paper  map,  its  overlays  of  acetate  covered 
with  marks  of  grease  pencils  wielded  by  the  staff  mem¬ 
bers  congregated  around  it.  Today  the  paper  map  has 
been  replaced  in  brigade  and  larger  headquarters  with  a 
digitized  map  projected  onto  a  large-screen  display.  The 
grease  pencil  has  become  a  mouse  that  officers  use  to 
draw  objects  or  select  pre-computed  overlays  from  a  pull¬ 
down  menu  of  options.  The  map  and  overlays  are  stored 


in  the  computer  as  data  structures,  are  processed  by 
algorithms  that  can  generate  in  seconds  products  it  would 
take  soldiers  many  hours  of  tedious  effort  to  duplicate, 
and  can  be  sent  instantly  to  relevant  consumers  anywhere 
on  the  Global  Information  Grid  (GIG). 

Advanced  automated  geospatial  tools  (AAGTs) 
transform  commercial  geographic  information  systems 
(GIS)  into  useful  military  services  for  Network  Centric 
Operations.  Because  of  their  basis  in  commercial  GIS, 
they  have  widespread  applicability  to  fire,  police,  disaster 
relief,  and  other  problems  characterized  by  a  command 
hierarchy.  The  advanced  situation  awareness  provided  by 
AAGTs  can  do  much  more  than  simply  speed  up  calcula¬ 
tions.  They  are  changing  the  way  military  operations 
are  conducted.  The  development  of  tools  is  shaped  by 
military  necessity,  but  the  decision  making  process  itself 
is  being  shaped  by  the  automated  tools  that  provide  warf¬ 
ighters  with  more  robust  situational  awareness. 

This  reality  of  21st  Century  Command  and  Control 
places  a  major  responsibility  on  researchers  who  develop 
tools  to  support  soldiers  as  they  perform  their  duties.  It  is 
essential  that  we  accurately  assess  the  value  of  the  tools 
we  develop  with  respect  to  planning  processes  and  situ¬ 
ation  awareness.  These  assessments  can  then  be  used  to 
shape  future  research  and  development  efforts. 

Intense  research  and  development  efforts  are  under¬ 
way  in  many  organizations,  funded  by  different  agencies, 
with  the  goal  of  moving  the  state  of  the  art  forward  and 
pushing  the  latest  generation  of  AAGTs  into  the  field  to 
meet  the  current  urgent  need.  A  rapid  development  and 
procurement  process  is  needed  if  we  are  to  provide  warf¬ 
ighters  with  tools  that  provide  force  multipliers  and  save 
lives. 

Research  indicates  that  sound  methodolgies  for  as¬ 
sessing  the  value  of  decision  support  tools  for  task  perfor¬ 
mance,  coupled  with  effective  development  processes  that 
make  use  of  the  feedback  thus  obtained,  can  dramatically 
improve  the  effectiveness  of  decision  support  (Adelman, 
1992;  Boehm,  et  al.,  1984;  Hicks  and  Hartson,  1993). 

This  paper  describes  a  project  underway  at  the  U.S. 
Army  Engineer  Research  and  Development  Center  to 
evaluate  the  value  added  to  military  decision  making 
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through  the  use  of  AAGTs.The  specific  AAGT  to  be 
evaluated  is  the  suite  of  Battlespace  Terrain  Reasoning 
and  Awareness  Battle  Command  (BTRA-BC)  Tools  (U.S. 
Army,  2003).  The  BTRA-BC  components  build  upon  a 
commercial  GIS  tool  (ArcInfo).The  BTRA-BC  tools  cre¬ 
ate  information  and  knowledge  products  that  empower 
soldiers  with  information  to  enhance  their  understand¬ 
ing  of  terrain  and  weather  as  it  impacts  their  functional 
responsibilities.  Participants  performed  terrain  analysis 
using  the  Army's  Digital  Topographic  Support  System 
(DTSS)  both  with  and  without  the  enhanced  capabilities 
provided  by  BTRA-BC.  DTSS  is  a  fielded  system  that 
provides  topographic  engineering  support  to  terrain  and 
topographic  technicians  as  they  assist  military  plan¬ 
ners  (Herrmann,  2002).  DTSS  provides  geospatial  data 
generation,  collection,  management,  information  process¬ 
ing  and  services.  The  BTRA-BC  capabilities  evaluated  in 
this  study  include  identification  of  obstacles,  production 
of  a  modified  Combined  Obstacles  Overlay  (MCOO),  and 
generation  of  mobility  corridors.  Our  experiments  will 
provide  essential  information  to  evaluate  the  contribu¬ 
tion  of  the  BTRA-BC  tools  in  particular,  and  AAGTs  in 
general,  to  enhance  the  military  decision  making  process. 

The  paper  is  organized  as  follows.  Section  2  de¬ 
scribes  the  overall  scope  of  our  research  program  and  the 
scope  of  our  initial  set  of  experiments.  Section  3  dis¬ 
cusses  the  primary  and  secondary  hypotheses  which  will 
be  examined.  Section  4  lays  out  the  design  of  the  experi¬ 
ments  and  the  reasoning  which  led  to  this  design.  Section 
5  discusses  the  computing  environment  which  will  be  used 
in  the  experiments.  Section  6  describes  the  metrics  used 
to  allow  us  to  quantify  the  results  of  each  trial.  Sections 
7  and  8  present  the  proposed  statistical  analysis  and  a 
brief  discussion  of  the  importance  of  evaluation  during 
development. 


2.  Scope  of  Experiments 

Our  ultimate  objective  is  to  evaluate  the  benefit  to 
commanders  at  the  brigade  level  and  below  of  combining 
a  fully  developed  AAGT  with  currently  available  Com¬ 
mand  and  Control  planning  tools.  The  scope  was  limited 
in  the  first  experiment,  and  will  expand  successively  in 
later  experiments.  The  experiment  with  which  this  paper 
is  concerned  is  limited  to  the  Intelligence  Preparation 
of  the  Battlefield  (IPB),  specifically  the  terrain  analysis 
portion  of  IPB. 

The  baseline  for  this  series  of  experiments  is  the  cur¬ 
rently  fielded  DTSS  suite  of  tools,  as  implemented  using 
ArcGIS  9.1.  The  DTSS  tool  suite  consists  of  a  package  of 
software  tools  used  to  generate  tactical  decision  aids  for 
producing  off-road  and  on-road  speed  products;  combined 


obstacle  overlays  (COOs);  shaded  time  distance;  masked/ 
visible  areas  for  observation;  and  fields  of  fire,  cover  and 
concealment,  obstacles,  key  terrain,  and  avenues  of  ap¬ 
proach. 

The  AAGT  under  evaluation  is  the  Battlespace  Terrain 
Reasoning  and  Awareness  Battle  Command  (BTRA- 
BC)  Tool.  Our  first  experiment  evaluates  the  most  cur¬ 
rent  version  of  BTRA-BC.  When  fully  developed,  it  will 
consist  of  six  information  generation  components  and  five 
decision  tools  addressing  terrain  effects.  Each  of  these 
components  utilizes  terrain  feature  data,  digital  elevation 
models,  and  information  about  tactics,  techniques  and 
system  performance.  BTRA-BC  information  generating 
components  produce  information  addressing:  (1)  observa¬ 
tion,  cover  and  concealment,  obstacles  and  mobility,  key 
terrain  and  avenues  of  approach  (OCOKA),  (2)  integrated 
products  defining  operational  positions  of  advantage,  (3) 
advanced  mobility  analysis,  (4)  digital  ground  and  air 
maneuver  potential  and  (5)  tactical  structures  relating 
information  produced  by  the  other  components.  Decision 
tools  support:  (1)  predictive  multi-criteria,  multi-objec¬ 
tive  maneuver,  and  logistical  route  analysis  for  ground 
platforms  and  forces,  (2)  predictive  sensor  performance 
(e.g.,  infrared  CIR],  millimeter-wave  [MMW],  seismic, 
and  acoustic),  (3)  situation  assessment  and  (4)  predictive 
threat  assessment 

BTRA-BC  research,  development,  products  and 
architectural  approach  are  designed  to  empower  the 
Joint  and  Future  Force's  Battle  Command,  and  Intelli¬ 
gence,  Surveillance,  and  Reconnaissance  (ISR)  processes 
and  systems  in  a  networked  force  structure  through  the 
incorporation  of  actionable  terrain  and  weather  informa¬ 
tion  and  tools. The  BTRA-BC  approach  is  wholly  consis¬ 
tent  with  the  Army's  Future  Combat  System's  (FCS's) 
System  of  Systems  and  the  Defense  Information  Systems 
Agency's  (DISA)  Network  Centric  Enterprise  Services 
concepts.  If  successful,  BTRA-BC  will  be  capable  of  ben¬ 
efiting  the  FCS  C4ISR  (Command,  Control,  Communica¬ 
tions,  Computers,  Intelligence,  Surveillance  and  Recon¬ 
naissance)  applique  and  the  Joint  Distributed  Common 
Ground  Station  family  of  ISR  systems. 


3.  Hypotheses 

In  order  to  evaluate  the  "value"  of  BTRA-BC,  we 
needed  to  establish  what  constitutes  "value"  with  respect 
to  military  decision  making.  Discussions  with  both  mili¬ 
tary  operational  planners  and  members  of  the  BTRA-BC 
development  team  clarified  the  areas  where  AAGTs  in 
general  and  BTRA-BC  in  particular  would  be  valuable  to 
the  military  decision  maker. The  first  and  most  obvious 
inherent  value  of  an  AAGT  is  in  its  ability  to  reduce  the 
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time  spent  generating  a  given  tactical  decision  product. 
Since  the  timeframe  available  to  military  decision  mak¬ 
ers  is  limited,  the  reduced  time  in  which  AAGTs  require 
to  produce  the  desired  output  can  free  up  time  for  a  more 
through  analysis  of  the  large  amount  of  data  available. 
This  more  complete  analysis  is  expected  to  result  in  a 
higher  quality  output  which  will  be  of  more  value  to  the 
decision  maker. 

The  second  value  contribution  of  an  AAGT  is  its 
automation.  Many  of  the  initial  tasks  traditionally  done 
by  terrain  analysts  with  paper  maps  are  sufficiently  rote 
in  nature  that  an  AAGT,  given  digital  information  and 
the  appropriate  parameters,  can  perform  these  functions 
more  quickly  and  with  less  error  than  a  human.  The  con¬ 
cern  that  automating  these  tasks  will  reduce  the  analyst's 
familiarity  with  the  terrain  and  understanding  of  the 
impact  on  the  military  planning  may  be  valid,  but  the  ex¬ 
perts  we  consulted  believe  that  the  automated  tasks  are 
procedural  and  not  analytical  and  that  using  the  output  of 
the  AAGT  will  not  compromise  the  analysis  of  the  data  or 
the  level  of  understanding  of  the  analyst.  The  experiment 
was  designed  to  test  this  prediction. 

It  follows  from  the  discussion  above  that  the  primary 
hypotheses  to  be  tested  in  the  initial  evaluation  are  the 
following:  In  comparison  with  analysts  using  currently 
available  tools,  we  hypothesize  that  trained  and  experi¬ 
enced  terrain  technicians  who  use  BTRA-BC  will: 

1.  Produce  certain  terrain-dependent  Intelligence 
Preparation  of  the  Battlefield  outputs  more 
quickly.  Rationale:  The  automation  in  BTRA- 
BC  should  allow  the  subjects  to  complete  the 
repetitive,  tedious,  and  rote  tasks  more  quickly. 

2.  Produce  a  higher  quality  output.  Rationale: 

The  automation  in  BTRA-BC  should  minimize 
errors  of  omission  and  in  calculation  and  stan¬ 
dardize  the  graphical  representation  of  impor 
tant  terrain  features. 

3.  Display  as  good  an  understanding  of  the  impact 
of  the  given  terrain  on  military  decision  making. 
Rationale:  The  judgment  required  to  complete 
the  required  tasks  will  still  be  required  when 
using  BTRA-BC. 

The  secondary  hypotheses  to  be  investigated  became 
apparent  as  the  determination  of  "value"  and  the  design 
of  the  experiment  evolved.  The  automation  of  some  tasks 
which  were  previously  performed  manually,  which  adds 
value  to  using  an  AAGT,  will  likely  reduce  the  varia¬ 
tion  in  the  output.  As  this  reduction  in  variation  does  not 
necessarily  add  value,  this  was  not  considered  a  primary 
hypothesis.  The  structure  of  the  experiments  requires  the 
repetition  of  various  tasks  and  there  was  concern  that  a 


learning  effect  might  skew  the  results  of  the  experiment. 
The  consensus  of  experts  in  terrain  analysis  indicated 
that  this  would  be  a  very  minor  effect  and  as  such  learn¬ 
ing  effects  are  considered  secondary  hypotheses.  The 
secondary  hypotheses  to  be  investigated  include: 

1.  The  output  generated  with  BTRA-BC  will  be 
more  uniform,  i.e.,  have  less  variance  in  the  first 
two  of  the  three  categories  above  (speed  and 
quality),  than  that  generated  without  the  use 

of  BTRA-BC.  Less  variation  in  the  output  when 
using  BTRA-BC  is  expected  due  to  the  level  of 
automation  incorporated  in  BTRA-BC. 

2.  There  will  not  be  a  learning  effect  due  to  the 
subjects  gaining  experience  with  the  C2  plan¬ 
ning  tools  during  the  first  trial.  The  subjects  have 
previous  training  and  extensive  experience  using 
the  C2  planning  environment  used  in  the 
experiment.  The  tasks  the  subjects  are  asked  to 
perform  are  those  that  they  normally  perform 
with  the  C2  planning  tools.  A  single  additional 
usage  will  not  be  evidenced  as  a  learning  effect. 

3.  There  will  be  little  or  no  learning  effect  due  to 
the  repetition  of  the  tasks  required  in  each  trial 
as  the  subjects  have  performed  similar  tasks 
numerous  times  prior  to  the  evaluation. 

4.  There  will  be  little  or  no  learning  effect  on  the 
subjects'  cognitive  process  due  to  using  BTRA- 
BC  in  the  first  trial.  BTRA-BC  automates  pro¬ 
cesses  with  which  the  subjects  are  extremely 
familiar  and  will  provide  no  additional  insight 
into  these  processes. 


4.  Study  Design 

The  general  study  design  will  remain  consistent 
throughout  the  series  of  experiments  and  will  employ  a 
within  subjects  design.  A  within  subjects  design  is  one  in 
which  each  subject  performs  both  sets  of  tasks.  A  within 
subjects  design  is  particularly  valuable  when  the  number 
of  available  subjects  is  limited,  as  in  the  current  case.  In 
addition,  the  results  from  the  sets  of  tasks  can  be  com¬ 
pared  for  each  subject  thus  eliminating  subject  specific 
effects  that  might  skew  or  add  variability  to  the  results. 
The  within  subjects  design  is  superior  to  the  between  sub¬ 
jects  design  (subjects  only  perform  one  task)  especially  if 
there  are  no  learning  effects. 

The  participants  performed  the  same  tasks  on  two 
similar  military  planning  scenarios,  where  one  of  the 
tasks  is  performed  with  BTRA-BC  functions  in  addition 
to  DTSS  functions  and  the  other  task  with  DTSS  func¬ 
tions  only.  The  two  trials  are  essentially  identical  except 
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for  the  use  of  BTRA-BC  in  addition  to  currently  deployed 
geospatial  tools.  The  order  of  the  tasks  was  randomly 
selected  so  that  half  of  the  subjects  perform  each  of  the 
tasks  first.  Randomizing  the  order  of  the  tasks  enables 
the  analysis  to  control  for  learning  effects. 

The  instructions,  tasks,  requested  outputs,  and  evalua¬ 
tion  of  these  outputs  were  the  same  in  both  trials  with  the 
exception  of  geographic  references  necessitated  by  the 
requirement  to  have  different  geographic  areas  for  each 
trial.  Different  geographic  areas  are  required  to  prevent 
participants  from  just  repeating  their  responses  from  the 
first  trial  when  they  form  responses  for  the  second  trial. 
The  two  areas  were  carefully  selected  for  their  geograph¬ 
ic  similarity  such  that  the  tasks  performed  by  the  partici¬ 
pants  and  the  expected  results  were  as  nearly  identical  as 
possible. 

The  participants,  Army  non-commissioned  officers 
trained  as  terrain  technicians,  were  split  into  four  groups 
that  are  evenly  balanced  as  to  the  experience  of  the 
participants.  Two  of  these  groups  performed  the  tasks 
first  without  BTRA-BC  and  then  with  BTRA-BC.  The 
other  two  groups  reversed  the  order  of  the  use  of  BTRA- 
BC.  Within  each  of  these  pairs  of  groups,  the  two  groups 
differed  on  which  scenario  was  used  on  the  first  day  and 
which  on  the  second  day. This  design  allows  the  analysis 
to  control  for  any  variance  in  results  due  to  the  experi¬ 
ence  of  the  subjects  and  the  order  of  the  sets  of  tasks  or 
the  experience  level  of  the  participants. 

The  tasks  consisted  of  that  portion  of  the  IPB  begin¬ 
ning  with  analyzing  the  specific  terrain  given  a  Combined 
Obstacle  Overlay  (COO)  up  to  the  point  of  generating 
potential  Avenues  of  Approach  (AA).  Specific  tasks 
included: 

1.  Identify  Mobility  Corridors  (MC) 

2.  Categorize  mobility  corridors  by  type  of  force 

3.  Group  mobility  corridors  to  form  potential  AAs 

4.  Identify  choke  points  on  potential  AAs 

The  participants  produced  a  graphic  overlay  depicting 
(1)  all  Mobility  Corridors  (MC),  (2)  four  potential  AAs, 
(3)  Choke  Points  on  potential  AAs,  (4)  recommended  AA, 
and  (5)  recommended  2nd  echelon  Areas  of  Operations 
(A00). 

Prior  to  beginning  the  tasks,  both  groups  of  par¬ 
ticipants  received  standardized  training  on  the  use  of 
BTRA-BC  and  the  DTSS  planning  tools  to  be  used  in  the 
experiment.  The  training  was  sufficient  to  perform  the 
required  tasks  given  the  subjects'  level  of  experience  with 
automated  systems  and  included  training  on  the  modes 
and  features  unique  to  BTRA-BC.  The  last  phase  of  the 


training  required  the  participants  to  perform  tasks  based 
on  the  training  and  similar  to  those  that  the  subjects 
would  encounter  during  the  trials,  but  of  lesser  complex¬ 
ity. 


5.  Environment 

The  evaluation  was  conducted  using  the  Digital 
Topographic  Support  System  (DTSS)  as  the  baseline 
technology.  DTSS  can  be  categorized  as  an  Automated 
Geospatial  tool  (AGT)  as  opposed  to  a  next  generation 
AAGT,  such  as  the  BTRA-BC  tool  we  are  evaluating. 
DTSS  is  the  currently  fielded  set  of  geospatial  tools  used 
to  support  the  IPB.  Modules  of  the  BTRA-BC  are  com¬ 
patible  with  DTSS  and  were  embedded  in  the  version  of 
DTSS  used  in  this  experiment.  During  the  trials  in  which 
the  subjects  are  using  BTRA-BC,  the  BTRA-BC  modules 
were  enabled.  During  the  trials  in  which  the  subjects  do 
not  use  BTRA-BC,  these  modules  were  disabled.  DTSS 
was  chosen  specifically  because  it  is  the  current  state  of 
the  art  in  AGTs,  it  is  currently  fielded,  and  the  potential 
subjects  are  familiar  with  its  use. 


6.  Metrics 

Because  trials  were  conducted  in  DTSS,  it  may  be 
possible  to  do  blind  scoring  for  some  of  the  hypotheses. 
The  criteria  for  evaluation  of  the  BTRA-BC  will  be  (1) 
a  comparison  of  the  rapidity  with  which  the  requested 
outputs  can  be  produced,  (2)  the  quality  of  those  outputs, 
and  (3)  the  level  of  understanding  of  the  participants  of 
the  impact  of  the  terrain  on  the  military  decision  making. 

The  evaluation  of  how  quickly  the  desired  outputs  are 
produced  can  be  measured  objectively  independently  of 
the  experimental  condition  by  logging  the  amount  of  time 
it  takes  participants  to  finish  the  task.  Thus,  blindness  is 
not  an  issue  for  the  speed  measure. 

The  evaluation  of  the  quality  of  the  outputs  may  be 
affected  by  the  evaluator's  ability  to  distinguish  between 
BTRA-BC  outputs  and  non-BTRA-BC  outputs.  If  the 
outputs  are  distinguishable  as  to  their  source,  then  the 
evaluation  of  quality  can  not  be  treated  as  blind.  We 
use  two  measures  of  quality  of  output:  the  first  consists 
of  objective  metrics  such  as  the  number  and  validity  of 
mobility  corridors  and  choke  points,  and  is  relatively 
independent  of  whether  blind  scoring  is  achievable.  The 
second  part  of  the  evaluation  of  quality,  which  is  still  in 
progress,  will  be  a  subjective  evaluation  by  impartial  ex¬ 
perts.  The  evaluators  will  judge  the  quality  of  the  output 
with  respect  to  the  usefulness  to  the  commander.  Criteria 
to  guide  the  evaluators  are  under  development.  Evalu¬ 
ators  will  provide  numerical  ratings  of  quality  of  output, 
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according  to  the  criteria  provided  to  them. 

To  evaluate  the  subjects'  understanding  of  the  impact 
of  the  specific  terrain  on  military  decision  making,  we  ad¬ 
ministered  a  questionnaire.  The  answers  to  the  questions 
were  not  outputs  of  DTSS  or  BTRA-BC.  The  answers 
required  judgment  and  reasoning  about  the  terrain  and  its 
effect  on  the  military  decision  making  not  just  regurgitat¬ 
ing  data  presented  by  DTSS  or  BTRA-BC. Topics  included 
in  the  questionnaire  included  justification  for  recommend¬ 
ed  AAs,  key  terrain  features,  cover  and  concealment,  an 
potential  locations  for  support  bases.  A  rubric  for  grading 
the  questionnaire  responses  is  under  development. 

The  assessment  of  the  second  and  third  primary  hy¬ 
potheses  and  the  secondary  hypotheses  was  augmented  by 
the  recorded  comments  of  observers.  A  statistical  analy¬ 
sis  will  be  conducted  to  test  each  of  these  hypotheses, 
and  the  combined  results  will  determine  the  evaluation  of 
the  value-added  of  BTRA-BC  versus  the  currently  fielded 
system. 


7.  Analyses  of  Results 

The  first  of  the  experiments  has  been  completed.  Re¬ 
sults  from  the  first  round  of  experiments  will  be  published 
in  a  later  version  of  this  paper  and  will  be  presented  later. 
Analysis  of  variance  will  be  performed  to  provide  quan¬ 
titative  estimates  of  the  degree  of  statistical  support  for 
(or  against)  the  hypotheses. 

8.  Discussion 

Evaluation  of  complex  systems  should  start  in  the 
development  phase  as  well  as  continuing  through  the 
procurement  process.  Evaluation  started  early  in  the 
development  process  allows  developers  to  choose  design 
options  with  the  greatest  potential  value  as  well  as  identi¬ 
fying  specific  problem  within  the  design.  It  is  well  known 
that  problems  found  early  in  the  design  and  development 
process  can  be  addressed  with  orders  of  magnitude  less 
impact  on  cost  than  if  they  are  not  identified  until  a 
later  stage.  Additionally,  evaluation  results  may  provide 
insights  into  possible  system  design  enhancements  not 
previously  identified.  A  key  factor  in  providing  the  best 
possible  feedback  to  the  designers  is  to  perform  evalu¬ 
ation  with  subjects  who  are  actual  users.  This  is  often 
somewhat  difficult  to  achieve  in  the  research  setting,  but 
true  "value"  can  only  be  determined  by  those  who  actu¬ 
ally  use  the  system. 

A  properly  designed  evaluation  program  can  build 
upon  initial  results  to  conduct  follow-on  evaluations  at 
each  step  in  the  design  process.  Once  the  value  of  the 


system  at  any  point  is  determined,  the  resulting  follow-on 
evaluations  can  be  easily  combined  with  previous  results 
to  form  a  coherent  overall  evaluation.  The  ultimate  goal 
of  the  design  and  procurement  process  is  the  fielding 
of  the  most  "valuable"  system  to  the  military  decision 
maker. 
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Effects  of  Age  and  Media  Packaging 
on  Designing  Training  Systems 

Alion  Science  and  Technology 

By:  I/I/.  Randall  Koons 


Abstract 

In  today's  work  place  there  are  four  distinct  generations 
working  side-by-side.  Generational  differences  can  be 
expressed  in  terms  of  visual  media  exposure.  There  are 
often  disconnects  between  the  Baby-boomers  and  the 
"Nintendo"  generations. Training  programs  designed  for 
learners  under  thirty,  are  often  conceived  and  funded  by 
decision-makers  who  are  over  fifty.  During  their  forma¬ 
tive  years,  these  two  groups  had  very  different  media 
exposure  that  shaped  their  expectations  for  the  packaging 
of  training  systems.  This  paper  will  explore  and  contrast 
generational  differences  that  result  from  exposure  to 
media  packaging  when  each  group  was  under  twenty;  and 
how  this  influences  the  design,  packaging,  and  funding  of 
training  programs  today.  For  the  under-thirty  learners,  ex¬ 
posure  to  an  expanded  array  of  media  options  has  evolved 
a  more  visual  and  kinesthetic  learning  style.  Understand¬ 
ing  these  changes  and  utilizing  techniques  from  the 
advertising  domain  should  improve  the  effectiveness  of 
current  and  future  training  programs.  Blending  modeling 
and  simulation  with  packaging  and  delivery  techniques 
used  in  advertising  and  entertainment  may  result  in  the 
cost-effective  creation  of  more  compelling  and  engaging 
training,  reducing  learning  time,  while  increasing  compre¬ 
hension  and  retention.  Developing  a  better  understanding 
of  these  generational  differences  is  key  in  the  design  and 
funding  of  training  systems. 


Introduction 

Effective  teachers  connect  with— engage— students  and 
create  a  more  conducive  learning  environment,  while 
making  the  material  more  easily  understood.  Test  scores 
may  vary  for  the  same  material  presented  by  different 
teachers  or  delivery  mechanisms,  which  infers  that  how 
well  the  learner  is  engaged  may  impact  the  ability  to 
absorb  and  recall  the  subject  material. 

This  paper  suggests  that  widely  different  expectations 
about  what's  engaging  are  affected  by  generational  dif¬ 
ferences  and  exposure  to  media  packaging.  Design  and 
deployment  of  effective  training  systems  must  acknowl¬ 
edge  these  influences  to  effectively  engage  the  target 


learner.  Advertising  creativity  and  media  packaging  exert 
a  powerful  influence  on  the  learner  generation,  which 
may  not  be  fully  understood  by  older  generations.  Meet¬ 
ing  learner  expectations— shaped  by  media— is  possible 
through  the  use  of  cost-effective  modeling  and  simulation 
(M&S)  technologies.  Packaging  training  with  familiar 
and  effective  media  techniques  will  better  engage  learn¬ 
ers  and  improve  results. 


Advertising  Creativity 

Advertising-style  creativity  can  address  the  issue  of  sub¬ 
jective  expectations. There's  a  link  between  training  and 
advertising.  They  share  three  common  elements:  compe¬ 
tition  for  attention,  conveying  a  message,  and  expected 
results.  Applying  advertising  creativity  may  increase  the 
success  of  a  training  system.  Advertising,  media,  and 
entertainment  creativity  have  influenced  expectations  of 
anyone  who  watches  TV  or  movies.  Often  subconscious, 
this  influence  results  from  carefully  packaged  media  de¬ 
signed  to  achieve  maximum  results.  Advertising  employs 
academic  analysis  and  focus  groups  to  improve  techniques 
and  the  effectiveness  of  the  business. 

In  advertising  theory  creativity  has  two  primary  deter¬ 
minants:  divergence  and  relevance.  Divergence  is  some¬ 
thing  new,  imaginative,  different,  or  unique.  A  divergent 
product— advertisement— must  fulfill  a  need  or  have 
relevance  to  the  target  audience.  Effectiveness  is  the 
measure  of  how  well  the  product  achieves  its  intended 
purpose,  which  is  dependent  on  social  recognition 
criteria— context.  An  ad  may  be  creative  to  one  group 
(e.g.,  senior  citizens)  and  not  to  another  (e.g.,  teenagers) 
(Smith  &  Xiaojing,  2004).  Therefore,  creativity  misap¬ 
plied  to  the  wrong  audience  will  likely  reduce  the  effec¬ 
tiveness  of  the  intended  effort  and  result  in  fewer  sales  of 
the  advertised  product. 

These  determinants  can  be  directly  applied  to  a  training 
program.  Divergence  is  used  as  the  hook  to  engage  and 
capture  audience  attention,  making  them  more  receptive 
to  the  message.  Relevance  maintains  their  attention- 
following  the  entire  message— and  will  likely  increase 
retention.  Effectiveness  is  measured  by  test  scores  and 
improved  practical  application  of  the  acquired  knowledge. 
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Understanding  the  contextual  determinant  of  the  target 
audience's  age  will  improve  divergence,  relevance,  and 
effectiveness. 


Generational  Cohorts 

Workplace  age-range  is  between  18  and  65.  In  2005 
this  spanned  four  distinct  generational  cohorts.  A  gener¬ 
ational— birth— cohort  is  "the  aggregation  of  individu¬ 
als  (within  some  population  definition)  who  experience 
the  same  event  within  the  same  time  interval"  (Ryder, 
1959).  Studies  define  some  cohort  classifications  based 
on  significant  historic  events,  such  as  the  Depression  and 
landing  on  the  moon.  Of  considerable  importance  is  when 
a  cohort  "comes  of  age"— early  adulthood  at  -20  years 
old.  Generally,  people  are  most  attached  to  memories 
and  experiences  from  their  early  adulthood.  Personal 
experiences  such  as  films,  television,  games,  and  adver¬ 
tising  tend  to  influence  people  well  into  their  adulthood 
(Schewe  &  Carlson,  2003)  (Kramer,  2004). 

Life  experience  affects  perceptions  and  expectations, 
while  influencing  what  is  considered  divergent  and 
relevant;  and  each  generation's  expectations  are  quite 
diverse.  Table  1  shows  an  overview  of  context  and  the 
attributes  of  the  four  generational  cohorts  in  the  2005 
workplace.  This  workplace  configuration  is  slowly  evolv¬ 
ing  as  WWII  and  Boomer  cohorts  retire. 

Media  Cohorts 

This  paper  will  address  the  influence  of  visual  media 
along  with  the  techniques  and  technology  that  deliver 
it.  Figure  1  shows  Birth  Cohort  social  context  and  what 
media  affected  the  different  ages.  Figure  1  demonstrates 
the  relationship  between  age,  media,  and  technology.  This 
figure  can  be  used  to  determine  a  person's  age  and  what 
media  and  technology  was  available  to  them  at  different 
ages  during  their  lives.  Television,  movies,  videogames, 
personal  computers,  and  other  technology  have  created 
another  set  of  media  cohorts.  A  selective  list  of  media  and 
technology  events  (y-axis)  has  been  arranged  by  the  year 
in  which  they  occurred.  Along  the  x-axis  is  year  and  age 
related  information.  The  Year  row  is  an  absolute  scale, 
against  which  ages  and  events  are  mapped.  Birth  Cohort 
section  shows  a  person's  age  in  2005,  which  can  be  read 
by  matching  age  with  the  year  row  above.  If  you  were 
50-years  old  in  2005,  you  were  born  in  1955.  Formative 
Years  section  maps  your  age  in  2005  to  the  years  when 
you  were  5-  and  20-years  old.  If  you  were  50  in  2005 
you  were  five  in  1960  and  20  in  1975.  Training  Partici¬ 
pants  shows  three  relative  work  roles  in  2005:  Sponsor, 
Designer,  and  Learner.  Age  and  Year  scales  will  provide 


mappings  to  media  and  technology  events.  The  cohort 
boundaries  become  apparent  in  Figure  1  as  three  Media 
cohorts  emerge:  Pre-Sesame  Street,  Sesame  Street,  and 
Nintendo  &  PC. 

Children's  Television  Workshop  (CTW)  developed  Sesame 
Street  to  teach  preschool  children  letters,  numbers,  and 
basic  reading  skills.  CTW's  design  was  based  on  the 
developmental  characteristics  of  their  target  audience. 
CTW  research  (Alexander,  2004)  (Gibson,  2004)  deter¬ 
mined  that  short  exciting  segments  of  information— col¬ 
orful  animated  visuals— capture  and  hold  the  child's  at¬ 
tention.  This  method  successfully  improved  the  audience's 
basic  letter  and  number  skills.  In  fact,  Sesame  Street 
reinforces  short  attention  span,  which  has  far-reaching 
consequences. 

Using  the  Formative  Years  scale  in  Figure  1,  it  can  be 
seen  that  Sesame  Street  had  practically  no  influence  on 
anyone  over  40.  Reading  and  associated  concentration 
along  with  fewer  media  choices  are  the  influences  on  the 
Pre-Sesame  Street  cohort.  The  innovative  technique  of 
packaging  instructional  and  other  television  media  define 
the  Sesame  Street  cohort  boundary.  It  began  a  transfor¬ 
mation  of  media  packaging  that  accelerated  the  pace  of 
information  packaging.  Nintendo  is  the  next  significant 
media  boundary.  Nintendo  and  the  PC  heralded  the  era 
of  interactive  entertainment.  Sesame  Street  is  television, 
a  passive  activity. The  Nintendo  cohort  actively  partici¬ 
pates  in  the  choice,  delivery,  and  outcome  of  their  media 
experience. 

The  Training  Participants  section  in  Figure  1  shows  three 
classifications  of  roles.  Sponsors  are  generally  between 
50-  and  60-years  old.  As  decision-makers  they  determine 
how  and  what  projects  will  be  funded.  Designers— gener¬ 
ally  between  30-  and  50-years  old— assess  learning  ob¬ 
jectives,  analyze  user  requirements,  and  develop  training 
systems. This  role  includes  Instructional  System  Design¬ 
ers  and  Subject  Matter  Experts  (SME).  Learners— typi¬ 
cally  between  18-  and  30-years  old— are  the  end-users  of 
training  systems. 

In  2005  sponsors  and  designers  over  forty  belong  to  the 
Pre-Sesame  Street  cohort  and  primarily  map  to  the 
Boomers.  Designers  under  forty  belong  to  the  Sesame 
Street  cohort  and  mainly  map  to  Gen-X.  Learners  are 
clearly  from  the  Sesame  Street  and  Nintendo  cohorts 
and  map  to  the  Gen-X  and  Millennial  cohorts.  This  is  a 
transitory  situation  that  will  change  in  20-years  when  few 
sponsors  of  the  Pre-Sesame  Street  cohort  will  be  work¬ 
ing. 

The  relationship  between  age,  work  responsibility, 
and  media  provides  a  context  to  better  understand  the 
influence  of  media.  However,  it's  important  to  consider 
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Cohort 

WWII 

Boomer 

Generation-X 

Millennial 

Year  Bom 

1922-1943 

1943-1969 

1969-1980 

1980-2000 

Description[1] 

•Rise  of  Middle  class, 
expected  prosperous  times 
to  continue  indefinitely 

•Viet  Nam,  Civil  Rights, 
Watergate,  oil  embargo,  less 
optimistic  about  financial  future 

•Free  agents  not  team 
players,  spirit  of 
entrepreneurship 

•More  idealistic, 
social-cause  oriented 

Coming  of  Age 
Historic  Events 
-20  years  old[1] 

•Post  WWII,  Korean 
conflict 

•Turmoil  of  the  1960s  and  first 
sustained  economic  downturn 
since  the  Great  Depression 

•Latchkey  children  of 
divorce,  cultural 
diversity 

•Information 
revolution,  Internet 

Coming  of  Age 
Media 

~20  years  old 

•Radio,  B&W  television, 
color  television 

•Star  Trek,  cable  television, 

VHS  VCR 

•MTV,  Nintendo, 

Windows  PC,  Internet, 
Jurassic  Park 

•DVD,  Broadband 
Internet,  Instant 
Messenger 

Learning  Style 
Preferences^] 

•Traditional  classroom, 
stable,  organized,  risk¬ 
free 

•Interactive  and  non¬ 
authoritarian,  relationship 
building 

•Self  directed,  learn- 
by-  doing  themselves 

•Teamwork, 
technology,  task 
oriented 

Table  1.  Historical  Generational  Cohorts 


cohort  expectations  as  trends  and  not  absolutes.  Based 
on  domain  experience  and  exposure,  some  sponsors  and 
designers  are  well  aware  of  learner  media  exposure  and 
expectations.  Cohort  generalizations  are  presented  in  this 
paper  to  set  context  for  this  subject. 


The  Influence  of  Media 

Sponsors  control  funding  of  training  systems  designed  for 
learners.  Sponsors  and  older  designers  have  significantly 
different  experiences  and  expectations  than  learners.  An¬ 
ecdotal  data  suggests  considerable  differences  between 
these  groups  regarding  effective  and  engaging  training. 
This  disconnect  may  be  due  to  formative-years  media  ex¬ 
posure.  Sponsors  grew  up  before  the  explosion  of  media 
and  communications  technology.  Learners  typically  don't 
remember  a  time  before  DVD,  cable  TV,  videogames,  and 
computers. 

A  sponsor,  who  was  55  in  2005,  was  20-years  old  in  1970 
and  their  media  exposure  had  been  primarily  limited  to 
radio  and  television;  and  TV  commercials  were  typically 
one-minute  long.  The  Pre-Sesame  Street  cohort  achieved 
management  positions  based  on  traditional  learning  styles 
and  methods  from  public  school  and  university  educations. 

After  1970— the  Sesame  Street  boundary— the  media 
world  changed  rapidly.  Figure  1  shows  an  accelerated 
expansion  of  media  and  technology  options.  Within  twenty 
years  the  increased  pace  resulted  in  the  PC  and  Nintendo. 
The  post  Sesame  Street  cohort's  formative  years  had 
more  media  options  utilizing  increased  pace  and  produc¬ 


tion  value,  which  shaped  their  expectations. 

Most  people  will  agree  the  pace  of  life  has  increased  and 
continues  to  accelerate.  Figure  1  shows  the  jump  in  me¬ 
dia  and  technology  options  and  a  dramatic  change  in  TV 
advertising.  The  Sesame  Street  model  demonstrated  that 
short,  fast-paced  media  packaging  was  very  effective.  The 
advertising  industry  adopted  similar  tactics  and  increased 
the  use  of  the  30-second  spot— commercial.  Some  twelve 
years  later  came  15-second  spots,  and  by  2003,  5-second 
spots  began  to  appear.  In  little  over  30-years,  one-minute 
of  TV  air  time  went  from  presenting  a  single  message  to 
the  possibility  of  four  to  twelve  different  messages. 

Not  only  have  commercials  grown  shorter,  but  the  number 
of  edits  per  spot— distinct  changes  in  the  visual  continu¬ 
ity— has  increased.  The  author  conducted  research  on  this 
aspect  by  downloading  295  TV  commercials  from  various 
Internet  sources.  These  commercials  covered  a  period 
from  1947  to  2005,  and  ranged  between  2-minutes  and 
15-seconds  in  duration.  This  is  by  no  means  intended  to 
be  an  exhaustive  study  of  commercials,  but  to  provide  a 
large  enough  sample  to  demonstrate  trends.  Table  2  uses 
161  30-second  spots  from  this  collection. The  number  of 
edits  was  determined  by  viewing  and  counting  the  total 
number  of  edits  per  spot.  Duration  of  edits  was  calculated 
by  dividing  the  average  number  of  edits  into  the  spot 
length  of  30-seconds. 

From  the  first  sample  period  to  the  last,  an  average  of 
5.7  edits  per  spot  has  jumped  to  20.2;  and  the  average 
number  of  seconds  per  edit  went  from  5.29  to  1.48. 

The  reversal  of  this  trend  in  the  70's  may  be  due  to  the 
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Figure  1.  Media  and  Technology  Events  Mapped  to  Generational  Cohorts 


limited  number  of  samples,  or  the  cost  of  editing  in  pre¬ 
digital  video  was  growing  too  expensive  per  edit. 


Period 

Sample 

s 

Avg.  #  of  Edits 

Avg. 

Sec./Edit 

’47-59 

6 

5.7 

5.29 

’60-69 

29 

12.2 

2.46 

70-79 

15 

9.7 

3.08 

’80-98 

14 

16.9 

1.77 

’01-05 

97 

20.2 

1.48 

Table  2. 30  sec.  spot  average  edits  and  edit  duration 


However,  the  overall  trend  shows  that  the  Pre-Sesame 
Street  cohort  was  exposed  an  average  edit  length  of  3.88 
seconds  compared  with  a  1.63  seconds  for  the  learner  co¬ 
hort;  presenting  more  information  to  absorb  in  a  shorter 
period.  Table  3  shows  how  ads  targeting  children— forma¬ 
tive  years— have  the  fastest  pace.  Sponsors  grew  up  with 


significantly  slower  media  pacing  than  learners.  These 
averages  were  derived  from  the  entire  sampling  regard¬ 
less  of  spot  length 


Period 

Avg.  edit/sec. 
Adults 

Avg.  edit/sec. 
Children 

’47-59 

7.84 

3.80 

’60-69 

4.30 

4.09 

70-79 

8.18 

2.50 

’80-98 

1.65 

2.29 

’01 -‘05 

2.08 

1.57 

Table  3.  Adults  va.children  media  pacing 


The  dramatic  increase  since  the  80's  is  not  simply 
stylistic,  but  a  result  of  digital  video  non-linear  editing 
introduced  in  the  mid  80's.  Word  processors  increased 
productivity  for  text  documents  and  non-linear  editing 
transformed  the  video  post-production  industry.  This 
revolution  enabled  editors  to  assemble  commercials  and 
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programs  in  less  time,  in  an  environment  that  charges 
editing  by  the  hour. 

In  1950  "Lazy-Bones"  was  the  first  available  TV  remote 
control.  In  2000,  more  than  99  percent  of  all  TVs  and 
100  percent  of  all  VCRs  and  DVD  players  sold  in  the 
United  States  are  equipped  with  remote  control  (Ze¬ 
nith).  When  a  viewer  watches  TV,  they  can  react  to 
boring  content— does  not  engage— with  the  touch  of  a 
button  and  begin  channel  surfing  for  content  that  will 
engage  the  viewer  to  stop  surfing.  The  remote  has  forced 
advertisers  to  apply  the  Sesame  Street  formula  and 
create  exciting,  fast-paced  content  to  reduce  boredom. 

It  must  be  fast  enough  to  engage  a  surfing  viewer  who 
spends  only  seconds  per  channel.  However,  it's  not  just 
increased  pacing  that  is  required  to  grab  attention,  but 
also  an  increase  in  production  value. 

The  advertising  and  motion  picture  industries  use 
production  value  to  apply  relative  quality  standards  for 
mass  media  delivered  products.  Production  value  can 
be  understood  by  comparing  science  fiction  film  special 
effects  over  the  past  60-years.  In  the  50's,  audiences 
accepted  spaceships  on  wires  with  rumbling  exhaust 
plumes  that  ignored  zero-Gs  in  the  vacuum  of  space. 

At  the  time,  that  was  considered  acceptable  production 
value,  but  not  in  2005.  "Star  Wars  Episode  III:  Revenge 
of  the  Sith"  (2005)  blurred  the  distinction  between  real¬ 
ity  and  effect— despite  rocket  engines  still  making  noise 
in  space.  However,  the  visual  quality  differences  between 
these  examples  demonstrate  the  concept  of  increased 
production  value. 

Advertisers  and  filmmakers  understand  the  necessity 
of  production  value.  "The  ultimate  judge  of  production 
value  is  the  target  audience.  If  you  keep  them  interested 
in  your  film,  it's  a  sign  of  good  production  value.  If  they 
come  back  for  your  next  film,  you  are  on  the  right  track." 
(Bitar,  2003).  The  same  is  true  of  TV  programs  and  com¬ 
mercials,  videogames,  and. ..training.  Learners,  exposed 
to  an  increased  level  of  production  value  and  pacing  has 
increased  their  expectations  of  computer-based  training 
(CBT),  which  is  considered  as  another  media  experi¬ 
ence.  Audiences  expect  fast-paced,  high  production 
value  media.  Advertisers  and  filmmakers  know  this  and 
spend  accordingly  on  TV  programs  and  commercials,  and 
videogames.  Table  4  shows  the  increased  investment  for 
the  Star  Wars  movies  due  to  competition  for  audience 
dollars  and  attention.  The  values  in  the  2005  $  column 
are  adjusted  for  inflation  by  Original  $/Factor. 

Technology  has  affected  the  media  production  industry 
with  greater  use  of  digital  effects.  Hollywood  films  now 
costs  an  average  of  $60-million,  doubled  since  1996 
(Kelleher,  2004).  More  effects  increase  production 


Movie  Title 

Yr. 

Original  $  m 

Factor 

[B] 

2005$ 

Star  Wars 

197 

7 

$12.5 

.310 

$40.3 

Empire  Strikes  Back 

198 

0 

$25 

.422 

$59.2 

Return  of  the  Jedi 

198 

3 

$32.5 

.510 

$63.7 

Phantom  Menace 

199 

9 

$115 

.853 

$134.8 

Attack  of  the  Clones 

200 

2 

$130 

.921 

$141.1 

Revenge  of  the  Sith 

200 

5 

$115 

1.00 

$115 

Table  4.  Star  Wars  production  costs  in  millions 


values,  which  further  raise  audience  expectations.  Media 
production  facilities  take  this  seriously  enough  that  tech¬ 
nology  improvement  is  like  an  arms  race.  "With  technol¬ 
ogy  squeezing  upgrade  cycles,  you  have  to  keep  buying  to 
stay  alive,"  says  Adi  Kishore,  a  Yankee  Group  entertain¬ 
ment  analyst  (Kelleher,  2004). 

Teenagers  are  watching  fewer  hours  of  TV!  Harris  Inter¬ 
active  and  Teenage  Research  Unlimited  polled  over  2,500 
teens  and  young  adults  (ages  13-24)  and  results  show  a 
weekly  average  of  16.7  hours  are  spent  online— exclud¬ 
ing  email— and  13.6  hours  watching  TV.  The  primary  rea¬ 
son  cited  for  this  trend:  the  ability  to  control,  personalize, 
and  manage  their  media  experience  and  content  (Silicon 
Valley/San  Jose  Business  Journal,  2003). The  use  of  PCs, 
videogames,  and  personalizing  music  selections  with  MP3 
players  delivers  far  more  control  of  media  options. The 
expectation  of  control  through  interactivity  is  another 
powerful  force  to  reckon  with  when  designing  effective 
training  systems. 

Learning  Styles 

In  1983  Dr.  Howard  Gardner,  professor  of  education  at 
Harvard  University,  developed  the  theory  of  multiple  in¬ 
telligences  (MI)  (Armstrong,  2000).  MI  postulates  there 
are  eight  different  learning  styles:  linguistic,  logical- 
mathematical,  visual-spatial,  musical,  interpersonal,  in¬ 
trapersonal,  naturalist,  and  bodily-kinesthetic. Traditional 
classroom  education  has  focused  primarily  on  the  linguis¬ 
tic  and  logical-mathematical.  Learners  from  the  Gen-X 
and  Millennial  cohorts— Table  1  — have  unique  learning 
style  preferences  and  worldviews;  they  prefer  visual  and 
kinesthetic  learning  styles.  Like  all  students,  they  learn 
more  effectively  when  taught  in  accordance  with  their 
learning  style  preferences  and  when  their  worldviews  are 


h  ttp://www.  dod-msiac.  org/ 


MSIAC  Journal  Volume  2,  Issue  1 0 


14 


Effects  of  Age  and  Media  Packaging  com. 


acknowledge.  Multitasking  is  an  attribute  of  the  learn¬ 
ers.  "If  people  are  really  good  at  processing  information 
from  lots  of  different  sources  and  you  don't  give  it  to 
them,  you  stifle  them,"  says  Marc  Prensky— CEO  and 
Founder  Games2Train  (Harmon,  1998).  Learn-by-doing 
is  an  effective  tool  for  training  the  learners,  as  it  takes 
into  account  their  learning  style.  Learners  have  been 
exposed  to  far  more  high  production  value,  fast-paced, 
interactive,  and  visually-based  content  than  the  sponsor 
cohort. 


Know  Your  Target  Audience 

Advertising,  motion  pictures,  TV,  and  videogame  pro¬ 
ducers  are  keenly  aware  that  successful  products  are 
designed  for  the  target  audience.  They  understand  the 
need  for  divergence  and  relevance;  pacing  and  pro¬ 
duction  value;  and  they  plan  and  invest  accordingly  to 
achieve  their  goal:  profit. Training  system  design  should 
be  no  different,  but  it  is.  "Death  by  PowerPoint"  is  a 
common  term  associated  with  training.  The  comment 
shouts  out  boring!  "Up  until  this  generation,  learning  has 
been  associated  with  pain,  but  that's  changing  because  a 
new  generation  doesn't  accept  it.  They  don't  buy  the  fact 
that  fun  and  work  have  to  be  different."  (Prensky,  2000) 
For  learners,  fun  is  often  associated  with  computers  and 
fast-paced  videogames  (Meyer,  1998). 

Age,  media  influence,  and  learning  styles  have  an  undeni¬ 
able  influence  on  personal  experience  and  expectations. 
Understanding  the  overall  effects  of  mass  market  media 
packaging  must  be  considered  when  developing  new 
training  systems.  What  works  for  one  generation  may 
not  work  for  another.  When  faced  with  funding  decisions, 
sponsors  may  wonder  why  the  concept  of  "fun"  should 
have  anything  to  do  with  training,  dismissing  concepts 
that  appear  to  be  fun  as  a  waste  of  money.  The  Pre-Ses¬ 
ame  cohort's  formative  years  were  shaped  by  traditional 
education,  which  focused  on  the  linguistic  and  logical- 
mathematical  learning  styles.  Multimedia  in  the  class¬ 
room  was  either  a  filmstrip  or  a  16mm  film  projector. 

Sponsors  may  ask,  "What's  wrong  with  the  younger 
generation?"  Learners  just  have  different  experiences, 
preferences,  and  learning  styles.  These  differences  must 
be  recognized  as  assets  that  should  be  cultivated  through 
proper  training,  which  connects  with  the  learner  for 
improved  training,  comprehension,  and  retention.  Yet  the 
generational  differences  between  sponsors,  designers, 
and  learners  may  prevent  the  development  of  successful 
training  systems.  Arguably,  the  primary  success-metric 
of  training  systems  is  cost/benefit.  Improved  productivity 
equals  profitability  and  safety  equals  reduced  expense. 
These  are  two  expected  results  from  training.  Withhold¬ 


ing  proper  investment  in  learner  appropriate  training  may 
save  money  in  the  short-term,  but  reduced  or  inefficient 
training  may  result  in  lower  productivity,  decreased 
safety,  and  a  net  long-term  loss. 


Entertain,  Engage,  and  Educate  (E3) 

If  training  design  is  to  benefit  from  applying  techniques 
from  advertising  theory,  consider  applying  E3.  Entertain 
is  the  first  E,  and  perhaps  for  some,  the  hardest  element 
to  reconcile  with  training. To  entertain  during  training  is 
not  intrinsically  negative.  Entertain  is  defined  as:  hold 
the  attention,  engage  or  engross  wholly.  Provide  an  initial 
dose  of  divergent  entertainment,  with  the  possibility  of 
more  during  the  training  experience,  may  serve  to  cap¬ 
ture  and  hold  the  attention  of  a  multitasker  and  improve 
retention.  Advertisers  and  CTW  craft  products  to  achieve 
success  by  using  this  method. 

Entertainment  yields  the  second  E:  Engage.  Allow¬ 
ing  learners  to  interact  with  training  systems  further 
reinforces  engagement.  Linear  page  turning  through  a 
PowerPoint  slide  show  is  minimally  more  interactive  than 
a  book.  The  Internet  and  videogames  have  ceded  control 
and  interactivity  to  the  learners.  Employing  learn-by-do- 
ing  and  the  visual-spatial  and  kinesthetic  learning  styles 
will  likely  improve  the  capacity  for  engaging  the  learner. 

Education— the  third  E— takes  place  when  the  learner 
is  fully  engaged  in  the  material.  Education  becomes  an 
easier  task  with  interactive  and  visual ly-rich  content.  This 
should  amplify  the  powerful  learning  advantages  of  CBT 
as  found  by  IBM  and  DoD  studies  showing  that  all  train¬ 
ees  retain  60%  of  material  when  taught  by  CBT  systems 
vs.  15%  in  standard  classrooms  (Garry,  1996). 

Attempts  have  been  made  to  improve  PowerPoint-based 
courses  by  adding  video  clips.  Perhaps  adding  any  sort  of 
dynamic  element  to  PowerPoint  may  be  an  improvement. 
That  must  be  the  logic  behind  using  some  of  the  many 
effects  to  reveal  text.  But  these  are  typically  the  efforts 
of  the  designer's  own  experiences  with  training  in  their 
formative  years.  It  is  in  fact,  a  significant  improvement 
on  the  filmstrip,  but  the  learners  see  right  through  these 
attempts.  Flying  text  and  dubious  quality  video  clips  are 
hardly  entertaining  to  the  Nintendo  generation;  no  E3 
here.  If  high  production  value  media  and  videogames  are 
the  experience  of  the  learners,  forcing  them  to  train  with 
PowerPoint  will  yield  less  than  optimal  results.  If  the 
goal  is  a  productive  and  safe  workforce,  it  will  be  more 
quickly  achieved  by  providing  familiar  venues.  However, 
a  balance  must  be  maintained  between  the  needs  of  the 
learner  and  the  reality  of  training  budgets. 

While  learners  may  be  technically  and  media  savvy,  they 
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generally  lack  the  experience  to  understand  the  mas¬ 
sive  investment  to  produce  a  videogame  that  costs  them 
fifty-dollars.  This  breeds  impatience  when  they  have 
direct  experience  with  an  inexpensive  product  and  can't 
understand  why  their  training  product  is  not  more  like 
the  game.  The  typical  look-and-feel  of  videogames  is  a 
dynamic,  interactive,  graphical  representation  of  real- 
world  environments.  Training  deployed  in  this  format  is 
familiar  and  readily  accepted  by  the  learners.  Planning 
and  designing  training  systems  in  this  venue  should  yield 
effective  E3  systems.  Given  the  alternatives  of  linear 
PowerPoint  vs.  real-time  interactive  graphics,  learners 
will  naturally  gravitate  towards  what  is  familiar  to  their 
experience  and  expectations. 

The  sponsor's  experience  with  interactive  graphics  may 
date  back  to  earlier  days  of  simulation  when  developing 
and  deploying  simulators  were  very  expensive.  Some  may 
argue  that  interactive  media  with  high  production  value  is 
not  practical.  Based  on  production  costs  of  films  and  vid¬ 
eogames,  they  may  conclude  that  high  production  value 
training  systems  are  cost  prohibitive. This  is  not  necessar¬ 
ily  so,  as  M&S  technology  may  provide  an  equitable  cost/ 
benefit  solution. 


Conclusions 

Developing  effective  training  systems  requires  under¬ 
standing  the  experience  and  expectations  of  the  target 
audience.  Sponsors  and  designers  have  different  experi¬ 
ences  and  expectations  shaped  in  part  by  a  different  pace 
and  quality  of  media  exposure. There  is  a  danger  of  map¬ 
ping  personal  experience  onto  another  group  who  doesn't 
share  the  same  experience.  The  advertising  and  motion 
picture  industry  understands  how  important  it  is  to  match 
product  development  with  the  target  audience.  To  do 
anything  less  reduces  the  odds  of  success  and  jeopardizes 
the  investment  in  product  development.  If  the  object  of 
training  is  to  improve  the  productivity  and  safety  of  the 
workforce,  that  goal  is  based  on  cost/benefit.  Investment 
in  training  is  not  altruistic,  but  profit  motivated:  reduce 
expenses  and  increase  profit.  Investment  in  training  sys¬ 
tems  that  are  based  on  the  experience  and  expectations 
of  sponsors  and  designers  will  lack  the  divergence  and 
relevancy  to  effectively  engage  the  learner  audience. 

The  Pre-Sesame  Street  cohorts  grew  up  at  a  time  with 
fewer  options  and  media  was  slower  and  less  sophisti¬ 
cated.  For  them  learning  was  often  a  painful  chore  and 
they  had  no  control  over  the  limited  media  venue. This  is 
almost  the  opposite  of  the  learner  experience.  The  pace 
and  production  value  of  media  that  the  learners  have 
grown  up  with  encourages  multitasking  and  interactiv¬ 
ity.  Training  systems  that  reflect  the  experience  of  the 


sponsors  and  designers  will  fall  woefully  short  of  learner 
expectations  and  reduce  the  effectiveness  of  the  system. 

Production  value  and  interactivity  come  with  a  price. 

Any  proposed  alternative  to  PowerPoint  training  must  be 
cost  competitive.  Learners  weaned  on  $50  videogames 
with  excellent  interactivity  and  high  production  value  are 
generally  unaware  of  the  multimil lion-dol lar  develop¬ 
ment  costs.  Despite  learner  expectations  fueled  by  media 
exposure,  it  doesn't  mean  training  systems  will  totally 
match  their  expectations.  Nor  can  sponsors  and  designers 
cite  high  production  costs  of  videogames  and  media,  and 
keep  feeding  increasingly  sophisticated  learners  boring 
PowerPoint  training.  A  workable  compromise  must  be 
established  between  the  expectations  of  the  learners  and 
the  cost  constraints  of  the  sponsors. 

The  advertising  and  motion  picture  industries  don't  leave 
success  to  chance.  Target  audience  focus  groups  view  TV 
commercial  concepts  and  prerelease  ads  to  ensure  the 
intended  message  will  be  divergent,  relevant,  and  effec¬ 
tive.  Motion  pictures  are  often  prescreened  to  selective 
audiences  to  ensure  expectations  are  met  and  promote 
success.  The  design  of  training  systems  could  benefit  from 
these  techniques.  Sponsors  and  designers  can't  ignore 
different  learning  styles  and  believe  that  entertaining  and 
engaging  content  is  frivolous  or  too  expensive.  Designing 
systems  that  reflect  the  learning  styles  and  expectations 
of  the  sponsor  and  designer  is  not  the  way  to  build  effec¬ 
tive  training  systems  for  learners.  Reduce  the  divergence 
and  relevance  of  the  system  and  reduce  effectiveness;  and 
jeopardize  investments  made  in  training  systems. 


The  Way  Forward 

The  first  step  is  recognition  that  there  is  a  difference 
between  sponsors,  designers,  and  learners. Then  carefully 
analyze  those  differences.  Make  use  of  focus  groups  and 
seek  learner  input  at  the  design  process;  not  when  the 
system  is  deployed.  Look  to  the  technologies  that  are  fa¬ 
miliar  and  relevant  to  the  learners.  M&S  can  be  used  as  a 
cost-effective  means  of  designing  and  fielding  E3  training 
systems  that  would  likely  improve  training  results.  An  ini¬ 
tial  investment  in  the  development  of  linking  simulations 
with  text-based  content  and  learning  management  sys¬ 
tems  is  a  required  start.  Next,  develop  the  GUIs  that  will 
enable  SMEs  and  instructors  to  easily  create  and  modify 
content  and  exercises,  which  will  provide  a  tremendous 
cost/benefit  and  improve  the  quality  of  training. 

Since  all  of  this  must  take  place  in  the  world  of  emerg¬ 
ing  training  standards,  there  is  a  need  to  address  the 
potential  constraints  of  SCORM  (Sharable  Content  Object 
Reference  Model)  compliant  distance  learning  systems 
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(DL).The  continued  extension  of  SCORM  standards  that 
effectively  address  simulation  and  the  installation  of 
client-side  API  components  are  critical  to  ensure  the  DL 
experience  will  better  match  learner  expectations.  WEB- 
based  DL  training  systems  will  need  to  provide  much  of 
the  experience  and  benefits  learners  expect  from  the 
WEB. 

It's  hoped  that  the  information  and  concepts  presented 
in  this  paper  will  promote  a  better  understanding  of  the 
impact  of  age  and  media  packaging.  Much  of  what  people 
experience  through  media  exposure  is  unconscious.  The 
people  behind  the  designs  and  techniques  have  methodi¬ 
cally  and  carefully  designed  their  products  to  achieve  the 
maximum  results  with  their  target  audience. The  train¬ 
ing  community  needs  to  make  a  more  conscious  effort 
to  utilize  these  techniques  and  design  for  the  learner. 
Make  this  effort,  promote  E3  with  your  audience,  and  the 
rewards  will  be  great. 
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Protean  Media: 

An  Approach  to  Gaming  and 
Understanding  Future  Complexity 


Proteus  Management  Group  USA 


By:  Bill  Wimbish 


The  United  States  Army  War  College  uses  the  acronym 
VUCA  to  characterize  the  strategic  environment  in  which 
we  must  operate.  VUCA  stands  for  volatility,  uncertainty, 
complexity  and  ambiguity.  Today's  threats  target  our 
vulnerabilities  and  seams,  and  often  employ  idiosyncratic 
methods  and  asymmetric  techniques  which,  although  they 
are  not  new  in  the  course  of  history,  are  being  exploited 
using  new  tools  by  super  empowered  groups  and  individu¬ 
als  who  hide,  adapt,  learn  and  strike  quickly  and  with 
precision.  How  do  we  learned  to  cope  with  volatility, 
uncertainty,  complexity  and  ambiguity  in  order  to  holisti¬ 
cally  analyze  our  vulnerabilities,  identify  these  threats 
and  determine  systemic  root  causes  of  national  security 
issues  and  then  apply  solutions  and  develop  strategies  that 
can  withstand  the  long  term  rigors  of  a  complex  intercon¬ 
nected  world?  In  other  words,  do  we  really  understand 
what  it  is  we  want  to  do  and  what  we  do  not  want  to 
have  happen?  If  we  can't  repeat  patterns  and  if  outcomes 
based  on  constant  inputs  are  not  constant  then  how  can 
we  anticipate  or  predict  what  is  on  the  future  horizon? 

Understanding  the  Complexity  of 
Human  Terrain 

Today,  scientists  use  Complexity  and  Chaos  theory  to  as¬ 
sist  in  explaining  the  properties  of  complex  adaptive  sys¬ 
tems,  relationships  and  natural  phenomena.  In  complexity 
theory,  behaviors  and  actions  occur  in  the  constantly 
changing,  interdependent  interrelationships  of  evolving 
systems.  The  most  elusive  and  complex  phenomena  are 
human  thought,  behavior  and  the  interactive  networks 
within  the  social  and  cognitive  domains  .  Often  we  see  the 
cognitive  thought  and  resultant  actions  of  our  adversaries 
as  irrational  or  illogical.  But  this  may  just  be  a  reflection 
of  our  own  Western  biases  and  they  may  be  quite  rational 
within  a  different  context  based  on  different  life  experi¬ 
ences,  social,  cultural  and  religion  norms  or  ideologies. 
How  do  we  model  or  map  this  form  of  complexity? 

The  Proteus  Management  Group  (PMG): 
Examining  Future  Complexity 

Understanding  this  area  to  be  a  key  and  critical  to  under¬ 


standing  and  meeting  the  challenges  of  a  complex  future 
geo-strategic  national  security  environment,  in  2005,  The 
National  Intelligence  University,  Office  of  the  Director  of 
National  Intelligence  established  a  Futures  "think  tank", 
the  Proteus  Management  Group  (PMG)  to  examine  ar¬ 
eas  of  future  complexity. This  organization  was  spawned 
by  an  ad  hoc  International  consortium  that  based  its 
efforts  on  a  revealing  and  provocative  intelligence  project 
called  Proteus. The  Proteus  project  originated  as  an 
advanced  future  concepts  research  initiative  at  the  U.S. 
National  Reconnaissance  Office  in  1999.  Interim  results 
of  the  work  were  published  in  a  book  entitled  Proteus 
Insights  from  2020.  The  book  contains  the  original  nine 
"insights"  and  five  "planes  of  influence"  that  were  de¬ 
rived  from  examining  five  "future  scenarios." 

Today  the  PMG  consists  of  nearly  300  national  and 
international  Fellows  and  Associates  across  the  many 
Government,  Department  of  Defense  (DoD),  intelligence, 
academic,  and  business  communities  who  are  examining 
and  sharing  "Proteus"  related  strategies  and  new  and 
emerging  processes  and  methods  of  analysis  and  decision¬ 
making  to  meet  21st  century  U.S.  national  security 
needs. 

The  PMG's  charter  is  to  promote  further  discourse,  study 
and  research,  focusing  on  the  refinement,  development 
and  application  of  Proteus  Insights  (PI)  and  other  PI  re¬ 
lated  new  and  emerging  "futures"  concepts,  methods,  pro¬ 
cesses  and  scenarios. The  group's  overall  intent  is  to  assist 
strategic  and  high-operational  level  decision  makers, 
planners  and  analysts  in  "outside  the  box"  consideration 
and  critical  analysis  of  National,  military  and  intelligence 
issues  within  the  Joint,  Interagency,  Intergovernmental 
and  Multinational  (JIIM)  environment  by  encouraging 
them  to  consider  differing  values,  frame  complex  issues 
holistically  and  scan  the  horizon  to  define  the  future  envi¬ 
ronment  in  order  to  identify  discreet  threats  and  capital¬ 
ize  on  hidden  opportunities. 

This  broad  charter  has  generated  a  number  of  initiatives 
to  pursue  the  group's  intent.  One  of  these  venues  entails 
examining  serious  gaming,  using  a  computer  assisted 
Role  Playing  Simulation  (RPS)  as  a  value-added  part  of 
the  suite  of  current  government  and  military  modeling 
and  simulation  (M&S)  tools  and  applications  which  are 
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designed  to  educate  and  inform  analysts,  planners  and 
decision  makers  about  future  complex  adaptive  systems; 
especially  in  the  area  of  human  terrain. 

The  following  discussion  will  first  briefly  look  at  the  cur¬ 
rent  and  near  term  M&S  efforts  in  this  area  and  their 
associated  limitations  and  challenges.  Second  it  will 
address  the  potential  of  serious  gaming  to  enhance  these 
initiatives  to  better  emulate  and  portray  the  most  elusive 
and  complex  phenomena;  human  thought,  behavior  and 
interactive  networks  within  the  social  and  cognitive  do¬ 
mains.  And  third,  it  will  highlight  the  operational  charac¬ 
teristics  of  the  Naval  Post  Graduate  School's  and  PMG's 
Protean  Media  Critical  Thinking  Game  and  examine  its 
future  potential. 


Modeling  and  Simulation  Challenges  in  the 
Future  Complex  Environment 

Since  9/11,  the  beginning  of  Global  War  on  Terrorism  and 
follow-on  combat  operations  in  Iraq  and  Afghanistan,  the 
U.S.  government  and  military  have  been  rapidly  trans¬ 
forming  their  emphasis  from  a  Cold  War  doctrine,  which 
focused  almost  solely  on  major  war;  e.g.  direct  combat 
operations,  kinetic  effects  and  large  conventional  battles, 
to  new  irregular  and  unrestricted  warfare  concepts  that 
not  only  include  conventional  or  kinetic  operations  but 
also  include  operations  that  addresses  the  reality  and 
complexity  of  the  human  dimension  of  conflict.  Winning 
the  peace  and  achieving  strategic  end  state  will  involve  a 
thorough  understanding  of  entity  (friend,  foe  or  neutral) 
belief  systems  (however  rational  or  irrational),  resultant 
direct  and/or  indirect  actions  and  the  interaction  and 
reaction  among  these  various  entities.  Within  the  Op¬ 
erational  Net  Assessment  (ONA)  concept,  the  military 
has  coined  a  holistic  framing  convention  and  acronym, 
PMESII  (Political,  Military,  Economic,  Social  Infrastruc¬ 
ture,  and  Information)  that  outlines  the  integrated  and 
interrelated  systems  and  domains  that  are  acted  upon  or 
are  affected  through  the  use  of  all  elements  of  national 
power:  Diplomatic,  Informational,  Military  and  Economic 
(DIME). 

The  future  challenge  for  the  M&S  community  as  a  whole 
is  not  developing  the  algorithms  to  model  or  simulate 
effects  of  physical  military  systems;  e.g.  Probability  of 
Kill  (PIO,  Correlation  of  Forces  and  Means  (COFM)  and 
other  force  on  force  models  (FoF).  But  the  real  chal¬ 
lenge  is  doing  this  for  psychological  effects  or  human 
aspects  within  the  complex  cognitive  and  social  domains. 
Although  attempts  have  been  made,  most  current  live,  vir¬ 
tual,  and  constructive  (L-V-C)  simulations  developed  by 
the  U.S.  military  to  educate  and  train  leaders  and  staffs 
have  not  been  able  to  fully  portray  or  account  for  cogni¬ 


tive  complex  adaptive  systems  where  rational  or  irratio¬ 
nal  thoughts,  ideas  and  actions  abound  and  relationships 
are  constantly  changing  and  morphing. 

For  example,  most  major  joint  and  service  oriented  simu¬ 
lations  used  currently  to  educate  intelligence  analysts, 
decision  makers  and  staffs  have  done  a  remarkable  job 
at  portraying  physical  or  kinetic  actions  and  resultant 
effects  on  physical  systems  of  multiple  entities  within  the 
battle  space  or  theater  of  war. These  include  the  Joint 
Forces  Command's  (USJFCOM)  Joint  Theater  Level 
Simulation  (JTLS),  which  is  a  large  interactive,  comput¬ 
er-assisted  simulation  that  replicates  a  theater  of  war  and 
incorporates  the  Joint  Conflict  and  Tactical  Simulation 
(JCATS)  program,  an  entity-level  interactive  simulation 
which  simulates  operations  in  urban  terrain.  Another 
simulation  of  this  genre  is  the  U.S.  Army's  Joint  Land 
Component  Constructive  Training  Capability  (JLCCTC) 
federate,  which  is  a  federation  of  current  and  developing 
training  simulations  which  stimulate  significant  asym¬ 
metric  activity,  However,  as  mentioned,  these  simulations 
have  not  yet  reach  their  full  potential  in  emulating  the 
changing  dynamics  of  political,  economic,  social  and  infor¬ 
mational  actions  and  resultant  effects. 


DoD  M&S  Efforts:  A  Holistic  Look  at  the 
Future  PMESII  Environment 

The  good  news  is,  that  the  Department  of  Defense  (DoD) 
and  its  multinational  partners  have  continued  to  move 
forward  to  establish  greater  corporate  M&S  oversight 
and  community  collaborative  effort  that  is  providing  guid¬ 
ance,  coordination  and  integration  of  future  live,  virtual, 
and  constructive  (L-V-C)  simulations  across  the  military 
services  and  Combatant  Commands.  This  ongoing  effort 
transcends  simulating  and  modeling  the  physical  aspect 
of  the  operational  environment  (i.e.  effects  of  weapon 
systems)  and  is  actively  promoting  new  M&S  develop¬ 
ments  to  improve  tools  for  educating,  planning,  analysis 
and  decision-making  that  reflect  all  PMESII  effects  of 
DIME  actions  through  a  closer  examination  of  agent 
based  modeling. 

For  example,  The  Army's  Joint  Land  Component  Con¬ 
structive  Training  Capability  (JLCCTC)  federate,  which 
stimulates  significant  asymmetric  activity,  is  using 
the  newly  developed  Joint  Non-kinetic  Effects  Model 
(JNEM)  to  determine  realistic  effects  of  economic, 
political,  religious  and  social  concerns  of  a  given  local 
population  through  modeling  civilian  reactions  to  military 
action.  Similarly,  another  non  kinetic  representation,  Syn¬ 
thetic  Environment  for  Analysis  and  Simulation  (SEAS), 
a  simulation  created  by  Purdue  University  and  used  by 
JFCOM,  simulates  an  impact  or  effect  on  target  local 


h  ttp://www.  dod-msiac.  org/ 


MSIAC  Journal  Volume  2,  Issue  1 0 


19 


Protean  Media  cont. 


population,  entity  leadership  as  well  as  infrastructure. 

The  Army's  Training  and  Doctrine  Command  (TRADOC), 
in  its  ongoing  futures  wargaming  and  experimentation 
effort,  Unified  Quest,  also  uses  a  series  of  simulations  and 
modeling  tools  within  the  Integrated  Gaming  System  that 
focus  on  emulating  the  human  terrain:  beliefs,  resultant 
actions;  social  networks,  direct  and  indirect  relationships 
between  entities  and  effectiveness  of  non-kinetic  military 
actions  in  future  complex  environments. 

The  Defense  Intelligence  Agency  has  funded  a  program 
called  Senturion,  a  predictive,  agents  based  model  and 
analysis  tool  that  simulates  the  bargaining  process  on 
political-military  issues. This  model  profiles  stakeholders 
and  their  political  decision  making  process.  Senturion 
prioritizes  and  sequences  well-established  theories  from 
microeconomics  and  political  theory  and  uses  input  from 
subject  matter  experts  and  rule  and  equation  based  algo¬ 
rithms  that  govern  stakeholders'  political  interactions  and 
employs  an  agent  based-game  theoretic  approach  that 
profiles  stakeholders  and  simulates  the  political  decision 
making  process. 

Overall,  DOD  M&S  initiatives  are  providing  analysts, 
planners  and  decisions  makers  with  a  markedly  greater 
appreciation  and  understanding  of  the  human  aspects 
of  rational  and  logical  actions  of  the  PMESII.  However, 
these  tools  are  still  limited  in  application  because  they 
are  generally  based  on  linear  and  deterministic  processes 
versus  stochastic  and  can  not  fully  account  for,  model  or 
simulate  the  cognitive  domain  or  human  nature  and  its  il¬ 
logical  or  irrational  or  counter-cultural  and  social  beliefs 
and  often  unpredictable  resultant  actions.  Being  able  to 
account  for  these  actions  are  especially  key  in  strate¬ 
gic  campaign  planning  or  policy  planning,  since  it  takes 
weeks  and  months  for  the  effects  to  manifest  themselves 
in  many  cases. The  challenge  is  to  set  model  conditions 
to  flag  and  recognize  effects,  and  be  able  to  exploit 
success  in  adequate  time  to  react  to  the  window  of  op¬ 
portunity.  So  what  additional  steps  can  the  government 
and  the  military  take  to  gain  additional  understanding 
and  insight  into  changing  dynamics  of  relationships  and 
discreet  interrelationships  within  or  between  complex 
adaptive  human  systems  in  order  to  prevent  unintended 
consequences  or  second  and  third  order  effects  that  this 
form  of  complexity  entails?  Serious  Gaming  agent  based 
applications  may  add  the  additional  tool  to  provide  the 
necessary  fidelity  needed  to  provide  additional  insight  to 
this  complexity  question. 


Serious  Gaming:  Adding  Fidelity  to 

Complexity  Modeling  and  Simulation 

Adding  robust  serious  gaming  using  computer  simulation 
and  computer  assisted  RPS  to  future  modeling  and  simu¬ 
lation  efforts  that  include  humans  or  agents  in  the  loop  to 
act  on  human  beliefs  and  has  the  potential  to  be  the  next 
step  in  providing  government  and  military  organizations 
with  the  needed  capability  and  fidelity  to  address  and  an¬ 
swer  these  questions.  If  used  properly,  serious  gaming  and 
RPS  can  be  used  as  a  laboratory  to  study  human  complex 
adaptive  systems,  especially  the  psychological,  social, 
and  cultural  aspects  that  are  needed  to  more  effectively 
conduct  future  experimentation,  educate  leaders  and 
assist  intelligence  analysts,  planners  and  decision  makers 
in  gaining  insight  and  addressing  the  challenges  of  the 
future  complex  PMESII  environment. 

Serious  gaming,  a  young  concept  and  capability  derived 
from  the  multibillion  dollar  entertainment  computer 
gaming  industry,  is  beginning  to  gain  significant  recogni¬ 
tion  especially  in  business  and  academia  as  a  potential 
strategy  and  educational  tool  for  addressing  future  com¬ 
plexity  and  gaining  knowledge  and  understanding  about 
complex  adaptive  systems.  Serious  Gaming  Conferences, 
symposia  and  programs,  like  the  2006  Serious  Gaming 
Summit  the  at  the  Woodrow  Wilson  Center's  Serious 
Gaming  Initiative  are  bringing  more  and  more  communi¬ 
ties  together  with  the  gaming  industry  and  developers  of 
games  like  Full  Spectrum  Warrior,  Civilization  III,  Linage 
and  SimCity. 

These  activities  are  promoting  fruitful  dialogues  and 
discussions  on  how  to  apply  serious  gaming  methodol¬ 
ogy  within  the  government,  military  and  privates  sectors. 
Games  like  Breakaway's  new  Incident  Commander,  a 
training  program  that  allows  cities  to  simulate  disasters, 
crises  and  even  terrorist  attacks  or  A  Force  More  Power¬ 
ful  (AFMP),  a  simulation  game  that  explores  the  strategy 
of  nonviolent  conflicts  against  dictators,  occupiers,  colo¬ 
nizers  and  corrupt  regimes  and  Peacemaker  developed 
by  graduate  students  at  Carnegie  Mellon  University's 
Entertainment  Technology  Center  that  simulates  the 
Mideast  conflict  and  calls  upon  players  to  negotiate  peace 
between  the  Israelis  and  Palestinians  are  all  examples  of 
games  with  potential  government  applications. 

Leading  research  professors  in  Video  Gaming  and  Learn¬ 
ing  at  The  University  of  Wisconsin's  Advanced  Distributive 
Learning  Co-Laboratory  assert  that  emulating  virtual 
worlds  through  gaming  is  the  most  powerful  learning  tool 
when  they  are  personally  meaningful,  experiential,  social 
and  epistemological  at  the  same  time.  They  further  pos¬ 
tulate  that  Gaming  lets  people  participate  in  new  worlds. 
Players  have  to  think,  talk  and  act  simultaneously  while 
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in  roles  that  are  usually  inaccessible  to  them.  In  virtual 
worlds,  participants  gain  situational  understanding  by 
experiencing  concrete  realities  across  multiple  contexts 
and  can  understand  complex  concepts  and  gain  situational 
understanding  without  losing  the  connection  between 
abstract  ideas  and  real  solvable  problems. 

Serious  Gaming  and  Potential 
Military  Applications 

The  U.S.  military  is  realizing  the  potential  of  serious 
gaming  and  is  currently  using  lower  level  tactical  training 
and  recruiting  applications  such  as  America's  Army.  An¬ 
other  project,  DARWARS,  part  of  the  Defense  Advanced 
Research  Project  Agency's  (DARPA)  Training  Superior¬ 
ity  Program  has  also  examined  new  kinds  of  cognitive 
experiential  educational  and  training  methods  for  units 
and  individuals  through  advance  simulation  and  gaming. 
DARPA  and  JFCOM  are  actively  expanding  their  search 
horizon  for  more  operational  and  strategic  military 
educational  and  analytical  applications  of  serious  gaming 
to  fill  the  gap  in  M&S  in  dealing  with  complex  and  often 
chaotic  PMESII  environments. These  applications  would 
serve  to  enhance  their  Integrated  Battle  Command  and 
Sentient  World  programs  as  well.  DARPA  has  specifically 
stated  that  "The  emphasis  in  the  Integrated  Battle  Com¬ 
mand  program  is  on  using  extant  modeling  technologies 
to  develop  full  scale  versions  of  a  set  of  models  that  will 
span  the  full  range  of  PMESII  systems  which  will  model 
these  systems,  and  the  behaviors  of  humans  involved  in 
these  systems,  and  their  response  to  DIME  actions,  to  the 
extent  that  they  generate  comprehensive,  authentic  pre¬ 
dictions  and  suggestions  that  will  augment  the  reasoning 
that  a  human  can  perform." 


The  "Far  Future  Conflict  Game"  is  another  bold  initiative 
to  use  serious  gaming  at  the  operational  and  strategic 
level  to  assist  in  understanding  complex  environments. 

The  game  is  being  co-developed  by  the  University  of 
Texas,  Dallas  and  U.S.  Army  TRADOC. This  complex, 
geo-strategic,  multiyear  game  is  designed  to  assist  future 
analysts  and  planners  to  examine  future  trends  and 
forecasts  to  determine  how  conflict  occurs,  why  it  occurs, 
and  the  specific  chain  of  events  that  lead  to  conflict.  The 
game  starts  in  a  current  virtual  world  with  a  wide  variety 
of  players.  In  this  virtual  complex  world,  players  repre¬ 
sent  individuals,  groups,  nation  states  to  include  terrorist/ 
criminal  organizations.  Gaming  conditions  constantly  shift 
based  on  player  decisions.  Decisions  are  required  in  the 
realms  of  sociology,  governance,  economics  and  resources, 
science,  technology  and  engineering,  military  develop¬ 


ments,  and  others.  The  game  is  more  than  kinetics;  it 
includes  military  conflict  as  well  as  natural  resource 
depletion,  starvation,  scientific  superiority,  civil  unrest 
and  wealth  disparity.  In  sum,  the  game  is  about  broad 
competition  across  a  number  of  realms. 


The  intelligence  community  is  also  exploring  Forterra's 
On-Line  Interactive  Virtual  Environment  (OLIVE)  tech¬ 
nology  platform.  Forterra  is  developing  an  operational 
system  through  which  intelligence  analysts  can  come 
together  to  test  out  and  prove  their  hypotheses  about  cur¬ 
rent  and  emerging  threats. The  system  will  bring  together 
experts  and  cultural/social  figures  from  all  over  the  world 
to  participate  and  role  play  in  an  on-line  system  to  deter¬ 
mine  the  validity  of  a  wide  variety  of  scenarios.  Analysts 
will  be  able  to  visualize  threats  and  play  them  out  in 
the  virtual  world  in  order  to  conduct  better  analysis  and 
ensure  multiple  threats  are  considered  and  acted  upon. 

Protean  Media:  A  Gaming  Approach  that 

Examines  Complex  Adaptive  Systems 

In  2001  the  Proteus  consortium  sponsored  the  develop¬ 
ment  of  an  educational  role  playing  simulation  that  allows 
participants  and  players  to  examine  the  results  of  human 
action,  interaction  and  subsequent  reactions,  convergence 
divergence,  conflict  and  agreement  within  a  complex 
adaptive  environment.  Entity  players  have  the  ability 
within  the  context  of  strategic  or  operational  events  or 
series  of  actions  (currently  contemporary  Iraq  is  mod¬ 
eled)  to  establish  goals  and  develop  their  own  strategies 
to  pursue  them.  Strategies  can  be  revealed  or  hidden  to 
achieve  these  goals  and  players  can  implement  each  one 
in  adjustable  real  time. 


Professor  John  Hiles  a  computer  scientist  and  career 
gaming  expert,  at  the  Naval  Post  Graduate  School's 
(NPS)  Modeling, Virtual  Environments,  and  Simulation 
(MOVES)  Institute  developed  the  Proteus  Media  RPS 
and  Critical  Thinking  Game.  It  is  a  "light"  and  low  cost 
RPS,  designed  to  model  shocked  or  stressed  complex  and 
adaptive  systems  and  naturally  evolving  events.  Its  tools' 
characteristics  are:  composite,  connecting  and  a  moving 
generation  system.  In  the  Game,  participants  face  am¬ 
biguous  complexity  manifested  by  others'  goals,  strategies 
and  intent.  Professor  Hiles' goal  is  "reification"  or  turn¬ 
ing  abstract  concepts  into  tangible  objects  to  handle  and 
manipulate.  This  Game  is  a  systems  approach  to  human 
conflict. The  speed  at  which  this  interaction  (virtual  and 
cognitive)  is  played  has  been  radically  increased  by  infer- 
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Protean  Media 

Game  Design  and  Play 

Wrapper:  Iraq  (8  Entities) 

^  I  •Stability  &  Reconstruction  _ \ 

•Tactical  -Strategic  — v 

Beliefs/Perceptions  Direct  or  Indirect  Actions 


Entity 

Shia  (2) 


Entity 

► 

Entity 

„  Coalition  g 

Insurgents 

Entity 

NGOs 


Entity 

4 - 

"  Entity 

V  Entity 

Iraq  Gov 

- ► 

Sunni 

4^- 

Shia  (1) 

Interaction/Relationships 


13  Entity  Actions 
•Hard  to  soft  I 

•Persuasive  to  Coercive 
•Multiple  action  modifiers 


Reactions 


•Meetings 

•Negotiations 


“PMESII  Effects” 


PME§II  System  Bfhavior  Modeis\ 
Social/ 
Iture 


Irregular^ 

MilitiP%  *  T  political/ 
^Groups*  Religious/ 


“DIME  Actions” 


mation  technologies  within  the  game.  The  Game  contains 
tables  of  modeled  behaviors  and  shows  the  interaction  of 
many  models.  Play  results  are  paradoxical  in  nature  and 
render  valuable  insights  about  the  cognitive  domain 

The  Game  demonstrates  complexity  by  taking  tacit  entity 
knowledge,  harnessing  it  and  letting  it  manifest  itself  as 
tumbling  reality.  Inputs  and  direct  or  indirect  actions  will 
often  not  produce  anticipated  or  expected  results  nor  will 
the  outcomes  be  readily  mapped  into  a  pattern  or  even 
consistently  repeated  based  on  the  volatile,  continually 
change  and  temporal  nature  of  human  relationships  and 
interactions  among  entities  and  populations. 

The  game  environment,  entities/groups/factions  and  their 
attributes  are  "wrappers"  and  can  be  changed  depending 
on  the  groups  or  regional  areas  where  conflict  or  signifi¬ 
cant  events  are  played  out  (disaster  relief,  humanitarian 
assistance,  regional  peacekeeping,  stability  &  reconstruc¬ 
tion,  civil  war,  etc).  The  game  as  currently  configured  rep¬ 
licates  the  factions/groups  within  Iraq  and  is  modeled  ac¬ 
cordingly.  The  game  is  oriented  around  thought,  interplay 
and  reality.  The  game  supports  thirteen  possible  entity 


player  actions  (hard  to  soft  and  persuasive  to  coercive) 
plus  a  meeting/negotiations  capability.  Each  action  has  a 
suite  of  modifiers.  The  key  objective  of  the  game  is  to  find 
where  factions  and  entities,  ideas  and  actions  converge 
and  either  conflict  or  agree  to  gain  additional  insight  on 
how  to  reestablish  provisional  order. 

Another  game  objective  is  to  directly  expose  players  to 
intended/unintended  consequences  generated  by  the  indi¬ 
rect  or  actions  and  decisions  of  the  various  entities.  The 
information  is  captured  in  the  data  and  by  incorporating  a 
thorough  After  Action  Review  process,  and  collecting  out¬ 
puts,  these  inferences  can  be  turned  into  lessons  learned 
supported  by  the  game  data  as  a  result  of  the  play. 

Protean  Media  Physical 

Gaming  Requirements 

The  software  is  not  proprietary  and  is  U.S.  Government 
owned. The  Protean  Media  game  is  Microsoft  based  and 
has  limited  graphics  making  it  portable  and  usable  on  al¬ 
most  any  government  laptop  or  desktop  computer.  Actual 
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Protean  Media  Sealable  "Wrappers” 

Capability  to  vaiy  and  scale  “wrappers,"  easily. 
Tactical  to  Strategic;  More  Entities  (25+) 


Coutries/  Regions  :  Africa,  ELropean/ELrasia .  East  Asia,  Soiih  Asia.  Near 
East,  Vtestem  Hemisphere  and  "Generic"  (Freedoni a) 

■Comdex  Crisis  Scenarios:  HA .  Disaster  Relirf,  PKSO,  Major  Vtor,  etc 


■Planes  of  Influence 
Actions 

■Relations  hips /Patterns 
■Complex  Characteristics 
■Finite 

Threats/Opport  unities 
■£''1/3**  Order  Effects 
■Unintended 
Consequences 


hardware  system  requirements  are  minimal  for  client 
and  server  systems.  The  client  server  machine  should  be 
at  least  a  Pentium  4  with  512  MD  RAM  and  Windows 
XP.  It  can  handle  up  to  25  clients  as  currently  configured. 
As  game  software  and  data  are  enhanced  or  additional 
clients  are  added,  it  may  require  additional  memory  and 
processing  speeds.  The  network  configuration  is  also 
relatively  simple  and  versatile.  It  can  be  run  on  a  non- 
secure  or  secure  LAN  or  wireless  network.  It  also  can  be 
web-based  and  played  over  the  internet. 


Protean  Media’s  Future  Potential 

The  current  Game's  basic  backbone  architecture  has 
great  growth  potential  to  optimize  play,  and  educational 
benefit,  not  only  within  the  military  but  the  government 
as  well.  The  Game  needs  to  be  reconfigurable  and  up 
graded  to  model/replicate  additional  scenarios,  entities 
and  multiple  planes  of  complexity.  It  can  add  more  than 
25  entities/players/actors  at  minimal  cost  with  only  minor 
design  modifications.  Additional  enhancements  to  the 
game  should  be  designed  not  only  to  look  at  cognitive 


interaction  and  convergence  but  to  better  understand 
the  results  of  unintended/intended  outcomes  or  second 
and  third  order  effects.  Although  the  Game  has  a  closed 
versus  an  open  architecture,  it  still  is  relatively  easy  to 
develop  a  series  of  different  "wrappers,"  tactical  to  stra¬ 
tegic,  within  different  countries  and  regions  that  can  be 
applied  to  most  real  world  or  future  scenarios  using  the 
basic  architecture  currently  on  hand.  Currently  the  NPS 
MOVES  Institute  is  examining  the  use  of  artificial  Intel¬ 
ligence  or  smart  agents  to  play  Game  entities  in  future 
versions. 

The  Game,  which  is  currently  configured  as  an  educa¬ 
tional  tool  to  reinforce  student  experiential  learning,  can 
also  be  redesigned  as  a  decision  support  tool  (expert 
agent  planning  or  a  rehearsal  vehicle,  or  outcome  model) 
for  future  experimentation  and  wargaming.  If  used  in  this 
venue,  the  validity  of  outcomes  should  be  based  on  deci¬ 
sions  and  actions  of  experts,  using  the  many  attributes 
particular  to  an  actor/group/entity.  The  latter  configura¬ 
tion  will  take  significant  effort  and  major  work  to  do 
adequately;  however,  it  is  feasible.  Lastly,  to  improve 
feedback  and  to  facilitate  data  collection  and  analysis, 


h  ttp://www.  dod-msiac.  org/ 


MSIAC  Journal  Volume  2,  Issue  1 0 


23 


Protean  Media  cont. 


Protean  Media  Data  Collection  and  Analysis 


Improved  Analytical  Function  for  player  or  assessor  review 


Identify,  Sort  and  Quantify: 

•Regional/Country  stability 
•Population  Support 

•Planes  of  Influence  Actions 
•Relationships,  Patterns,  Threads 
•Finite  Threats/Opportunities 
•2nd/3rd  Order  Effects 
•Unintended  Consequences 


/  Assessment/ 
AAR  Information! 

•Actions/Effects: 
•What 
•When 
•Where 
•Why 


the  game  needs  a  process  to  gather,  sort  and  parse  key  in¬ 
formation  about  entity  actions,  relationships  and  resultant 
reactions  and  second  and  third  order  effects. 


Summary 

Although  there  has  been  little  emphasis  on  the  use  of  Se¬ 
rious  Gaming  within  the  JIIM  community  in  the  past,  the 
PMG  effort  and  others  are  significant  first  steps  to  begin 
capitalizing  on  a  largely  untapped  capability  to  supple¬ 
ment  the  current  L-V-C  simulations  discussed  above  in 
order  to  more  effectively  conduct  experimentation  and 
intelligence  analysis,  test  strategy  and  educated  leaders 
a  every  level  about  the  complex  environment  where  they 
will  operate.  However,  more  research  and  experimenta¬ 
tion  is  needed  to  convert  commercial  serious  gaming  ap¬ 
plications  to  meet  and  address  the  gamut  of  government 
and  military  operational  and  educational  requirements. 


The  Protean  Media  is  not  the  total  panacea  for  gaming 
or  modeling  complexity;  however,  it  provides  a  founda¬ 
tion  for  others  to  build  upon.  The  ultimate  goal  of  the 


PMG  is  to  develop  a  scalable  variable  wrapper,  agent 
based  interactive  experiential  education  and  planning  and 
implementing  tool/game  which  can  capture  entity  many 
actions,  develop  threads  and  relationships  and  identify 
cascading  second  and  third  order  effects  and  unintended 
consequences  in  complex  environments. This  will  be 
done  by  incorporating  the  complex,  temporal  and  chang¬ 
ing  effects  of  human-behavior/belief  systems  and  social/ 
cultural  dimensions.  The  game  must  continue  to  integrate 
advances  from  gaming  theory,  belief  systems/influence 
models,  perception  and  cognitive  modeling  and  other 
complex-nonlinear  programming  efforts  to  create  the 
ultimate  "paradoxical"  game. 


Point  of  Contact: 


Bill  Wimbish 

William.wimbish@us.army.mil 
Proteus  Management  Group  USA 
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